Lexicon and Grammar in Probabilistic Tagging of Written English

The paper describes the development of software for automatic grammatical ana]ysi$ of unl~'Ui~, unedited English text at the Unit for Compm= Research on the Ev~li~h Language (UCREL) at the Univet~ of Lancaster. The work is ~n'nmtly funded by IBM and carried out in collaboration with colleagues at IBM UK (W'~) and IBM Yorktown Heights. The paper will focus on the lexicon component of the word raging system, the UCREL grammar, the datal~zlks of parsed sentences, and the tools that have been written to support developmem of these comlm~ems. ~ wozk has applications to speech technology, sl~lfing conectim, end other areas of natural lmlguage pngessil~ ~y, our goal is to provide a language model using transin'ca statistics to di.~.nbigu~ al.:mative 1~ for a speech .:a~nicim device. 1. Text Corpora Historically, the use of text corpora to provide mnp/ncal data for tes~g gramm.~e.al theories has been regarded as important to varying degn~es by philologists and linguists of differing pe~msions. The use of co~us citations in ~-~,~ma~ and dictionaries pre~t~ electronic da~a processing (Brown. 1984: 34). While most of the generative 8r~- ,-a,iam of the 60S and 70S ignored corpus ant,,: the inc~tsed power Of the new t~mlogy ,wenlw.l~ points the way to new applications of computerized text cmlxEa in dictiona~ makln~_: style checking and speech w, cognition. Compmer corpora present the computational linguist with the diversity and complexity of real language which is more challenging for testing language models than intuitively derived examples. Ultimately grammatl must be judged by their ability to contend with the teal facts of language and not just basic constructs extrapolated by grammm/ans. 2. Word Tagging The system devised for automatic word tagging or part of speech selection for processing nmn/ng Enfli~ text, known as the Constituent-Likelihood Automatic Word-tagging System (CLAWS) (Garside et aL, 1987) serves as the basis for the current work. The word tagging system is an automated c~mponent of the probabilist/c parsing system we are curnmtly woddng on. In won/tagging, each of the rurmi.$ words in the coqms text to be processed is associated with a pre-termina/ symbol, denoting word class. In e.~enc~ the CLAWS suite can be conceplually divided imo two phases: tag assignment and tag selection.